Verifying Equivalence of Spark Programs Technical Report 1-Nov-2016

Spark is a popular framework for writing large scale data processing applications. Our goal is to develop tools for reasoning about Spark programs. This is challenging because Spark programs combine database-like relational algebraic operations and aggregate operations with User Defined Functions (UDF s). We present the first technique for verifying the equivalence of Spark programs. We model Spark as a programming language whose semantics imitates Relational Algebra queries (with aggregations) over bags (multisets) and allows for UDFs expressible in Presburger Arithmetics. While the problem of checking equivalence is undecidable in general, we present a sound technique for verifying the equivalence of an interesting class of Spark programs, and show that it is complete under certain restrictions. We implemented our technique in a prototype tool, and used it to verify the equivalence of a few small, but intricate, open-source Spark programs.